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p\| , Abstract 
> ; 

. We derive the limiting matrix kernels for the the Gaussian Orthogonal and Sym- 

\ plectic ensembles scaled at the edge, with proofs of convergence in the operator norms 

lO ' that assure convegence of the determinants. 

o ■ 

^ ■ 

O ■ I. Introduction 

^ I For a large class of finite determinantal processes the limiting distribution, as ^ oo, 

+^ ■ of the right-most "particle" is expressible as the Fredholm determinant of an operator Kxuy 

a with kernel 

L- ■ — ^-^ ^ ' = A(x + z) Aiy + z) dz, 

i ; ... - . aI; l... J,, n, ....... ... 

I results are proved by first establishing that for finite the distribution of the the right-most 
particle is the Fredholm determinant of an operator K^. The limit theorem then follows 
once one proves K]\f — > XAiry in trace norm. The classic example is the finite Gaussian 
Unitary Ensemble (GUE) where is the Hermite kernel [6] and the limit A^ — >■ cxd is 
the edge scaling limit of the largest eigenvalue. In this case the unitary invariance of the 
underlying probability measure is manifest. 

As is well known one can also consider the edge scaling of the largest eigenvalue for 
the Gaussian Orthogonal Ensemble (GOE) or the Gaussian Symplectic Ensemble (GSE) [9]. 
These limiting distributions are now widely believed to describe the universal behavior of 
a right-most particle for a large class of processes under additional symmetry restrictions 
[3, 8]. In the orthogonal and symplectic ensembles, the distribution of the largest eigenvalue 
is the square root of a Fredholm determinant of an operator whose kernel is now a 2 x 2 
matrix kernel (of course, different iT^v for different ensembles). In [9] the derivation of the 
limiting distributions for the largest eigenvalue did not use limiting kernels, as it did for the 
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ensembles scaled in the bulk. Instead, differential equations determining the the finite N 
distributions were obtained first and then a limiting argument was applied to these. 

The edge-scaled kernels are already in the literature [2, 1], but as far as we know limit 
theorems in the operator norms that give convergence of the determinants are not. Because 
of the current interest in these ensembles, it seems useful to state and prove such theorems 
for the matrix kernels. For GSE we shall establish convergence in trace norm to a limiting 
operator Kqse- The GOE case is more awkward and for that we shall use weighted Hilbert 
spaces and an extension of the determinant involving the regularized 2-determinant, and 
convergence to /Cqoe will be in a combination of Hilbert-Schmidt and trace norms. These 
will give convergence of the determinants. 



II. The finite N kernels 



We shall follow the notation of [10], more or less. For both finite ensembles we denote 
by K]\f{x, y) the matrix kernel (and the corresponding operator) such that the expected 
value of nfc(l + /(Afc)) is given by 

E (n(l + /(A.))) = Vdet {I + Kr,f). (2.1) 

Here / is a "general" function and in the determinant it denotes multiplication by /. In 
particular the probability of finding no eigenvalues in a set J is given by 



det^I-K^Xj). (2.2) 

Here are the formulas for Kjsi. We set e{x) = ^sgnx, denote by e the operator with 
kernel s{x — y), and denote by the harmonic oscillator wave functions. 

For GSE, with weight function and N odd, we define 



iV-l 



Sn{x, y) =J2 ^n{x) <^n{y) + V ^ ^N-i{x) e^N{y), 

n=0 * ^ 



N-1 



ISN{x,y) = Yl ^'Pnix) <f„{y) + V ^ £^jv-i(a;) e<pN{y), 

n=0 * ^ 



Af-1 



SND{x,y) = - ^ ipn{x) if'^{y) - \/ — ^n-i{x) (pNiy)- 

n=0 



Then 



KNix,y) = ^ 



2 

/ SN{x,y) SND{x,y) 
\ ISN{x,y) SN{y,x) 



(2.3) 
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For GOE, with weight function e ^^/^ and N even we define SN{x,y), ISN{x,y) and 
S^Dix, y) by exactly the same formulas and then 



SN{,x,y) 



SND{x,y) 



) 



KN{x,y) 



(2.4) 



\ ISN{x,y) - e{x - y) SN{y,x) 



Despite their apparent similarity the kernels for GSE and GOE have very different prop- 
erties. First, the functions e(pn, which do not vanish at ±00 when n is even, make it appear 
that the entries of K]sr{x, y) do not vanish at ±00, but they are actually exponentially small 
for GSE. (See derivation in Section 8 of [10].) Thus the / in (2.1) can be quite general and 
even grow at ±00, and Kjsff is a trace class operator on © L^. 

This is not the case for GOE. First, the entries do not all vanish at ±00; second, the 
discontinuity in s{x — y) prevents K^f from being a trace class operator. (The discussion 
in Section 9 of [10] skirted these issues.) To take care of these problems we use weighted 
LP' spaces and a generalization of the determinant. Let p be any weight function such that 
e L} and such that all (pn £ -^^(p)-^ Fo^' any such p the matrix is a Hilbert- 
Schmidt operator on L^(p) ©L^(p~^),^ and this is the space on which acts. The diagonal 
entries of are finite rank, hence trace class. Now the definition of determinant extends to 
Hilbert-Schmidt operator matrices T with trace class diagonal entries by setting det(/ — T) = 
det2(/ — T) e"*"^^, where trT denotes the sum of the traces of the diagonal entries of T and 
det2 is the regularized 2-dctcrminant.'^ It follows from an identity for 2-determinants ([4], 
p. 169) that for this extended definition we still have the relation 



Using this fact, and defining det (/ — K^f) in this way,*^ the discussion of [10] can be carried 
through for any / G L°°. 



Here we follow the notation of Section VII of [9] and denote our scaling transformation 
by T, so that 



We shall consider only the limit of (2.2). We denote here by J any measurable set which is 
bounded below and replace J in (2.2) by r(J). We shall compute the matrix kernel Kqse 
such that the limit of the determinant in (2.2) as — > 00 is equal to det (J — KgseXj)- 

iRecall that L^ip) = {h : J \h{x)\^ p{x)dx < 00}. 

^That £ is a Hilbert-Schmidt operator from L'^{p) to L'^{p~^) is equivalent to p~^ G . 

^If T is a Hilbert-Schmidt operator with eigenvalues Hk then det2 (/ — T) = Y\{1 — p,k) e'"'- See [4], Sec. 



^The so-defined det (/ — Kjsif) is independent of the choice of p, since eigenfunctions corresponding to 
nonzero eigenvalues belong to Li^ip) © L'^{p~^) for all permissible p. In fact it is given by the usual Predholm 
expansion, modified for matrix kernels. We prefer the operator definition since it will enable us to establish 
convergence of the determinants more easily. 



det (7 - Ti) (7 - T2) = det (7 - Ti) det (7 - T2). 



III. Scaling GSE at the edge 




IV.2. 
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Observe first that the determinant is unchanged if the kernel Kn{x, y) XT{j){y) is replaced 

by 

t'Kn{t{x), r{y))Xj{y), 

where r' = 2~^/^A^~^/^. It is also unchanged if the upper-right entry is multiplied by r' and 
the lower-left entry divided by r'. (We shall use the same notation for the modified kernel.) 
Thus it is a matter of determining the limit, in trace norm, of the entries of the modified 
matrix kernel. 

If we write 
and set 

N-l 
n=0 

then 

Sn{x, y) = S%{x, y) + il^i^x) eip{y). 
Formula (57) of [9] is in our notation 

;>oo 

Slf{x, y) = [(p{x + z) ipiy + z) + i;{x + z) (p{y + z)] dz. 
Jo 

Define (fir — (for and ipr — ip o t. Then the substitution z r'z in the integral shows that 

t' S%{t{x), T{y)) ^t' / [ifir{x + Z)'4)r{y + Z)+Ipr{x + Z)(fir{y + Z)\dz. 

Jo 

It follows from results on the asymptotics of Hermite polynomials that 

lim t' (fir{x) = lim t' ipr{x) = —^A{x) (3.5) 

pointwise, where A{x) denotes the Airy function Ai(a;), and that there are estimates 

r'^rix) = 0(e-^), r'iJrix) = 0(6"^) (3.6) 

which hold uniformly in TV and for x bounded below. (There is a better bound but this one 
is good enough for our purposes. See [7], p. 403.) We shall show that this implies that if 



KAhy{x, y)= I A{x + z) Aijj + z) dz, 

J 



then 

T'Sl{T{x),T{y))^K^,y{x, y) 
in trace norm on any space L^(s, oo). 
To show that 

T fir{x + Z) l/jriy + z) dz ^ -KAiry{x, y), 
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(the other half of r' S%{t{x), T{y)) is treated similarly) we write the difference as 

r[T'ipr{,x+z)-2-^/^A{x+z)] r'i^riy+z) dz+ r 2-^/^A{x+z) [rV^(|/+^)-2-V2A(y+z)] dz. 
Jo JO 

Each summand is an integral over z of rank one kernels, and the trace norm of an integral 
is at most the integral of the trace norms. Thus the trace norm of the first summand is at 
most 

/ ||rVr(^ + z)- 2-'/^A{x + z)\\2 Wr'My + dz, (3.7) 
Jo 

where the first norm is taken with respect to the x variable and the second with respect 
to the y variable. It follows from (3.6) that the second norm is 0{e~^) uniformly in N, and 
from (3.6) and (3.5) that the first norm is also 0{e~^) uniformly in and tends pointwise 
to zero. Thus the integral has limit zero. The same argument applies to the othe integral. 

Thus S% scales to i^Airy and it remains to consider t' ipri^) (£V')r(?/), the last summand 
in t' Sn{t{x), T{y)). Of course t' il^r{x) -^Ai^x) in trace norm. Define 

c^ = - ip{x) dx, = - / i^{x) dx, 



roo poo 

$t(x) = / T'(j)r{z)dz, *r(a:) = / T'ilJr{z)dz. 

Jx Jx 



We have 



roo roo 

{^v)T{y) = - / v{z) dz^ - T'(pr{z) dz = -$(y). 

JT{y) Jy 

(Since TV is odd so is </?, so c<^ = 0.) Prom (3.5) and (3.6) it follows that this converges to 



1 

71 



A{z) dz 



in the norm of any space L'^{s, oo). 
Thus we have shown that 

r' Sn{t{x), T{y)) Kxuy{.x, y) - - A{x) / A{z) dz 

Z Jy 

in the trace norm of operators on any space LP'{s, oo). 

Next, observe that SnD{x, y) = —dySi^{x, y) and so (recall that we multiply the upper- 
right corner by r') 

t'^ Sj,D{t{x), T{y)) = -dyT'S:,{T{x), T{y)). 
Since the limiting relation we found above for Sn holds even after taking dy we see that 

t'^SnD{t{x), T{y)) ^ -dyKA^ix, y) - \a{x) A{y) 
in trace norm as A'^ ^ oo. 
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If we recall that we divide the lower-left corner by r' we see that it remains to find the 
limit of 

ISj,{t{x), T{y)) = eS%{r{x), T{y)) + ei^{r{x)) s^{T{y)). 
The last term equals 

-(c^-*^(x))$,(y). 
If we make the substitution z t'z in the integral for sSj^{r{x), T{y)) it becomes 

/ [{e^r){x + z)T'^r{y + z) + {etl:)r{x + z)T'ipr{y + z)]dz. (3.8) 

Replacing ei^r and S'^^r by what they are in terms of and gives 

roo 

- / [^r{x + Z) r'lpriy + z) + '^r{x + z) t' ^riv + z)] dz + C^^riv)- 

Jo 

Putting these together we obtain 

;>oo 

ISN(r(x), T{y)) = - / [^r{x + z) r'lPriy + z) + + z) r'ifiriy + z)] dz + ^r(x) ^r(y)- 

JO 

Now we use (3.5) and (3.6) as before, and deduce that 

roo poo poo 

ISN{r{x), T{y)) - / Ka{z, y) dz + - / A{z) dz ■ / A{z) dz 

Jx ^ Jx Jy 



in trace norm. 
Thus if we set 



1 

S{x, y) = KAi,y{x, y)-- A{x) / A{z) dz, 

I Jy 

SD{x, y) = -dyK^ryix, y) - \a{x) ^(y), 



1 

iA{z,y) dz + 

then we have shown that for GSE 



/•CO 2 roo roo 

IS{x, y) = - / Ka{z, y)dz+- / A{z) dz ■ / A{z) dz, 

Jx Z Jx Jy 



t'Kn{t{x), T{y)) Kq^e{x, y) := ^ 



/ S{x,y) SD{x,y) 
\lS{x,y) S{y,x) 



in trace norm as iV — > cxd. 

In particular, the probability that no eigenvalue lies in r(J) tends to 



^det{I -KgseXj) 

as A'" — > oo. 
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IV. Scaling GOE at the edge 

Before computing limits let us sec what the norm of a rank one kernel u{x)v{y) is when 
thought of as taking a space L'^{pi) to a space L^(p2)- The operator, denoted hj u^v, takes 
a function h e L'^ipi) to u{v, h), and so its norm is the L\p2) norm of u times the norm of 
V in the space dual to I/^(pi), which is LF'{pi^). Thus 

\\u v\\ = \\u\\l2^p^-) (4.9) 

We determine the limit of the scaled kernel 

T'K^{T{x),T{y)) 

where K]^ is now given by (2.4). The space on which this acts is L'^{p) © L?'{p~^) with any 
fixed permissible p which we now assume has at most polynomial growth at +oo.^ As before, 
we multiply the upper-right corner by r' and divide the lower-left corner by r', but do not 
change notation. 

For t'S%{t{x), T{y)), the main part of t'Sn{t{x), T{y)), consider the analogue of (3.7). 
Since this kernel takes I/^(p) to itself (4.9) gives for the analogue here 

Wr'ifrix + z)- 2-^/^A{x + z)\\l2^p) {{r'^y + dz. (4.10) 

The second factor is 0{e~^) uniformly in N since r''0r(y -\- z) — 0{e~^) and p~^ e L^. The 
first factor is at most the square root of 

/•oo 

J Ir'^rix + z)- 2-^1'^ A[x + z)\^p{x) dx 

for some s. Using (3.5) and (3.6) and the fact that p has at most polynomial growth we see 
that this is uniformly 0(e~^/^) and converges to zero pointwise. Thus (4.10) has hmit zero. 

The other integral arising in the aymptotics of r' S^{t{x)., ^iu)) is similar, and we deduce 
that 

t' S%{r{x), T{y)) KAiry{x, y) 

in trace norm, as an operator on L'^{p)- It remains to consider r' -0^(3:) (£(/?) ^(y). First, 
t' il^T^x) -^A{x) in the space L'^{p), again by (3.5) and (3.6) and the fact that p has at 
most polynomial growth. Now 

and since is even for GOE so is ip and c<^ ^ 0. In fact ^ ^ as N ^ oo and so we see 
that 

1 1 

^All functions are thought of as defined on some interval (s, oo). 
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uniformly, and so in the space L^(p Thus, 

t' Sn{t{x), T{y)) Km^{x, y) + ]^A{x) (l - A{z) dz^ 

in trace norm as an operator on L'^{p). 

Consider now the upper-right corner of the modified matrix, 

t''^ SnD{t{x), r{y)) = -dyr' Swirix), T{y)) - r'^iprix) (Priv), 

whose hmit we want to compute as an operator from L'^{p~^) to L^{p)- In the apphcation 
of (4.9) both norms are that of L'^{p)- In the verification that 

-dyT'S^^{r{x), T{y)) -dyK^i^y{x, y) 

we obtain an integral of the form (4.10) where the second factor in the integrand is replaced 

by 

l|TVUy + ^)||L2(p). 

This is bounded uniformly in and z because (3.6) holds also even after taking derivatives. 
The first integrand is estimated as before, so the integral tends to zero in trace norm. 

We also have t'iPt{x) ■;^A{x) and T'(pT{y) ■;^A{y) in L'^{p), and so we find that 

t'^ Sr,D{T{x), T{y)) ^ -dK^iryix, y)-^ A{x) A{y) 

in trace norm as before, but now as operators from L'^{p^^) to L'^{p)- 

For the lower-left corner, an operator now from L'^{p) to L?'{p~^) so in (4.9) both norms 
are L'^{p~^) norms, we have as before 

IS^{t{x), T{y)) = eS%{T{x), r(y)) + ei;{T{x)) £^(r(t/)). 

The last term equals 

Replacing e(fr and eif^r in (3.8) by what they are in terms of and "ifr now gives 

POO 

- / [^r{x + Z) rVr(y + Z)+ ^r{x + z) t' ifr^V + z)] dz + C^^ M , 

Jo 

SO we obtain now the identity 

/•oo 

ISN{r{x), r{y)) = - / [$^(x + z) r>^(y + z) + *^(a: + z) r'<pr{y + z)] dz 

Jo 

^r{y) - ^r{x) (C^ - My))- 

Now we deduce that 

PCX) 1 / /"^ /'°° r'^ \ 

ISj,(r(x),T(y))^-J^ KA(z,y)dz+-i^J^ A(z)dz + j^ A{z)dz-j^ A{z)dz^ 
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as trace class operators from -L^(p) to LF'{p''^). 

Finally, s{t{x) — T{y)) — e{x — y), so this is unchanged. 
Thus if we set 

S{x, y) = i^Airy(a^, y) + \ A{x) (l - A{z) dz^ , 

SD{x, y) = -dyKAiry{x, y) - ]^A{x) A{y), 

IS{x, y)^-J^ Ka{z, y)dz + - ^j^ A{z) dz + A{z) dz ■ A{z) dzj , 
then we have shown that for GOE 

S{x,y) SD{x,y) 



t'Kn{t{x), T{y)) KcoEix, y) :-- 



IS{x,y) - s{x -y) S{y,x) 



as A'^ — > oo in the sense that the kernels converge in the Hilbert-Schmidt norm of operators 
on L'^{p) ® L'^{p^^) and the diagonals converge in trace norm. (In fact all entries converge 
in trace norm, except for the fixed Hilbert-Schmidt operator e.) 

In particular, it follows from the continuity of the 2-determinant in Hilbert-Schmidt norm 
and the trace in trace norm that if the set J is bounded below then the probability that no 
eigenvalue lies in t(J) tends to 

^det(J-/s:GOEXj) 

as A?" — > oo. 
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